Papers by Shay B Cohen
Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics (2026.findings-acl)
Copied to clipboard
| Challenge: | unified vision–language models (VLMs) struggle to generate physically plausible transitions between frames from instructions. |
| Approach: | They find that VLMs struggle to generate physically plausible transitions between frames from instructions. |
| Outcome: | The proposed model outperforms state-of-the-art image editing models on Aurora-Bench . it achieves the best average human evaluation across all subsets of Aurora-bench compared with other models . |
PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing personalization benchmarks focus on chit-chat, non-conversational tasks, or narrow domains, failing to capture complexities of personalized task-oriented assistance. |
| Approach: | They propose a benchmark to evaluate personalization in task-oriented AI assistants . the benchmark features user profiles equipped with rich preferences and interaction histories . |
| Outcome: | The proposed benchmark features user profiles equipped with rich preferences and interaction histories . it also features a judge agent and user agent that employs the LLM-as-a-Judge paradigm . |
CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Mathematical reasoning remains a significant challenge for large language models (LLMs), despite advances in prompting techniques such as Chain-of-Thought (CoT). |
| Approach: | They propose a framework that enhances reasoning through two stages: Symbolic Conversion and Reasoning Execution. |
| Outcome: | The proposed framework outperforms traditional CoT on six out of seven benchmarks across four LLMs. |
PiCSAR: Probabilistic Confidence Selection and Ranking for Reasoning Chains (2026.findings-acl)
Copied to clipboard
Joshua Ong Jun Leang, Zheng Zhao, Aryo Pradipta Gema, Sohee Yang, Wai-Chung Kwan, Xuanli He, Wenda Li, Pasquale Minervini, Eleonora Giunchiglia, Shay B Cohen
| Challenge: | Recent studies show that large reasoning models (LLMs) achieve strong performance on complex reasoning tasks. |
| Approach: | They propose a method that scores each candidate generation using the joint log-likelihood of the reasoning and final answer. |
| Outcome: | The proposed method outperforms baselines with 2x fewer samples in 20 out of 25 comparisons. |
Theorem Prover as a Judge for Synthetic Data Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that large language models are increasingly capable of tackling mathematical problems. |
| Approach: | They propose an approach that iteratively refines theorem prover formalisation to mitigate errors. |
| Outcome: | The proposed method increases execution rate on the Lean prover from 60% to 87%, while human annotation is replaced with theorem prover feedback. |
Thinking in Schemas: Robust Syllogistic Reasoning in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | syllogistic reasoning models often mistake what sounds true for what is formally valid . content effect is a limitation of sluggish reasoning, which can lead to invalid conclusions . eisape et al., 2024: a key open problem for formal inference in natural language. |
| Approach: | They propose a schema-guided framework that disentangles semantic plausibility from logical validity. |
| Outcome: | The proposed framework outperforms existing frameworks while reducing CE. |
Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks like LOFT often overestimate LCLM performance by providing overly simplified contexts. |
| Approach: | They propose to use retrieval-attention-probing to filter and de-noise long contexts during decoding and joint retrieval head training alongside the generation head to improve LCLM performance. |
| Outcome: | The proposed approach outperforms RAG and GPT-4-Turbo on most tasks despite being a much smaller model. |
Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibility (2026.findings-acl)
Copied to clipboard
| Challenge: | a recent development of code-LLMs has demonstrated remarkable performance across various software engineering applications. |
| Approach: | They propose a round-trip code execution reasoning task to test round- trip consistency . they use zero-shot prompting, supervised fine-tuning on execution traces and self-reflection mechanisms to evaluate models . |
| Outcome: | The proposed benchmarks show that LLMs struggle with round-trip consistency . the benchmarks lack the internal coherence required for trustworthy code reasoning . |
Iterative Multilingual Spectral Attribute Erasure (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for debiasing are unable to exploit this opportunity because they operate on individual languages. |
| Approach: | They propose to iterate multilingual spectral attribute error (IMSAE) to mitigate joint bias subspaces across multiple languages through iterative SVD-based truncation. |
| Outcome: | The proposed method outperforms monolingual and cross-lingual approaches while maintaining model utility. |
One More Question is Enough, Expert Question Decomposition (EQD) Model for Domain Quantitative Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Domain-specific quantitative reasoning remains a challenge for large language models . we propose an approach to balance domain knowledge with computational efficiency . |
| Approach: | They propose an approach to balance domain knowledge with computational efficiency . it uses a two-step fine-tuning framework and a reward function to measure sub-questions' effectiveness . |
| Outcome: | The proposed approach outperforms state-of-the-art domain-tuned models and advanced prompting strategies in the financial domain. |
What can Large Language Models Capture about Code Functional Equivalence? (2025.findings-naacl)
Copied to clipboard
| Challenge: | SeqCoBench is a benchmark to assess how Code-LLMs can capture code semantics. |
| Approach: | They propose a benchmark to assess how Code-LLMs capture code semantics . they use seqCoBench to evaluate whether they can discern semantically equivalent or different pairs of programs . |
| Outcome: | The proposed benchmarks show that they can capture code semantics better than classical match-based retrieval scores. |